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Executive Summary 

Charter schools — public schools that are exempt from many of the procedural regulations that apply to 
regular public schools — are a widespread but poorly-studied form of education reform. With nearly 2,700 
charter schools now educating more than 684,000 children nationwide, policymakers and parents need to 
know how the education charter schools provide compares to that provided by regular public schools. 

Assessing the academic performance of charter schools is difficult, because many charter schools are tar- 
geted toward specific populations such as at-risk students, disabled students, and juvenile delinquents. 
This makes it very challenging for researchers to draw a fair comparison — comparing targeted charter 
schools to regular public schools is like comparing apples and zebras. As a result, there are very few reli- 
able research findings on the academic quality of charter schools as compared to regular public schools. 

This is the first national empirical study of charter schools that compares apples to apples — that is, test scores 
at charter schools and regular public schools serving similar student populations. By comparing "imtargeted" 
charter schools serving the general population to their closest neighboring regular public schools, we can 
draw a fair comparison and get an accurate picture of how well charter schools are performing. 

Measuring test score improvements in eleven states over a one-year period, this study finds that charter 
schools serving the general student population outperformed nearby regular public schools on math tests 
by 0.08 standard deviations, equivalent to a benefit of 3 percentile points for a student starting at the 50th 
percentile. These charter schools also outperformed nearby regular public schools on reading tests by 0.04 
standard deviations, equal to a benefit of 2 percentile points for a student starting at the 50th percentile. 

The study's strongest results came in Florida and Texas. In Texas, charter schools achieved year-to-year math 
score improvements 0.18 standard deviations higher than those of comparable regular public schools, and 
reading score improvements 0.19 standard deviations higher. These benefits are equivalent to 7 and 8 percen- 
tile points, respectively, from the 50th percentile. Florida charter schools achieved year-to-year math and 
reading score improvements that were each 0.15 standard deviations greater than those of nearby regular 
public schools, equivalent to a gain of 6 percentile points for a student starting at the 50th percentile. 
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Introduction 

Charter schools — public schools that are exempt 
from many of fhe procedural regulafions fhaf apply 
fo regular public schools — are a widespread buf 
poorly-studied form of educafion reform. Because 
charfer schools provoke less political resistance than 
other kinds of school choice, fhey have spread across 
fhe counfry rapidly; there are now nearly 2,700 char- 
ter schools spread across 39 states and the District of 
Columbia and educafing more than 684,000 stu- 
dents.^ With the size and rapid growth of the char- 
ter school movement, policymakers and researchers 
need to know whether the education provided by 
charter schools is superior to that available in tradi- 
tional public schools. 

But assessing the academic performance of charter 
schools is difficult, because many charter schools 
serve specifically targefed populations such as af- 
risk studenfs, disabled studenfs, and juvenile delin- 
quents. This makes it very difficult for researchers 
fo draw a fair comparison between charter schools 
and regular public schools; simply comparing the 
performance of regular public school studenfs to the 
performance of charfer school students is like com- 
paring apples and zebras. As a result, there are very 
few reliable research findings on fhe academic qual- 
ify of charter schools as compared to regular public 
schools. 

A comparison between charter schools and regular 
public schools that all serve general student popula- 
tions, however, would be fair — a comparison of 
apples fo apples. Using fhis approach, we found thaf 
imtargefed charter schools — that is, those that serve 
general student populations — perform moderafely 
better than neighboring regular public schools. Our 
overall national analysis found thaf over a one-year 
time period unfargefed charter schools outper- 
formed nearby public schools on math tests by 0.08 
standard deviations, which is the equivalent of a 



benefif of 3 percentile points for a studenf sfarfing af 
fhe 50th percentile. We also found fhaf imfargeted 
charter schools outperformed nearby public schools 
on reading fesfs by 0.04 sfandard deviafions, equal 
fo a benefif of 2 percenfile points from the 50th per- 
centile. Both these results were statistically signifi- 
cant at a high level. In individual states, we found 
inconclusive results in Arizona, California, and 
Norfh Carolina, and sfatistically significanf posifive 
resulfs in Florida and Texas. Some slates may have 
had inconclusive results because of small sample 
sizes. Our sfrongesf results, in Texas, foimd that char- 
ter schools achieved year-to-year score improve- 
ments of 0.18 sfandard deviafions relafive to 
comparable public schools on standardized math 
tests, and 0.19 standard deviations on reading tests. 
These benefits are equivalent to 7 and 8 percentile 
points, respectively, from fhe 50fh percenfile. 

Previous Research 

Charfer schools are public schools thaf operafe ouf- 
side fhe normal bureaucratic and regulafory school 
system. They are created by special agreement be- 
tween the school itself and an aufhorizrng agency. 
Stale laws differ regarding whaf insfifufions can 
aufhorize charter schools; in some states only local 
school boards can do so, while in other states the 
state department of educafion or stale universities 
can also serve as aufhorizing agencies. In all slates 
with charter school laws, however, the agreement 
between the school and the authorizer, called the 
school's "charter," lays out the terms by which the 
school is to be run. Charter schools are schools of 
choice; sfudents may choose to go to a charter school 
instead of the regular public school to which they 
are assigned. In exchange for freedom from many of 
fhe usual rules and regulations public schools must 
follow (for example, rules covering the hiring and 
firing of teachers), charter schools can be held di- 
rectly accountable for fheir performance by their 
authorizers. 
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Despite the rapid growth of charter schools, rela- 
tively little is known about whether charter schools 
perform better than regular public schools. One of 
the most important difficulties in studying charter 
schools is that many of them are targeted specifi- 
cally at particular student populations, and thus 
serve dramatically different kinds of students than 
regular public schools do. Although most states re- 
quire charter schools to have open enrollment poli- 
cies, charter schools can still target specific 
populations by describing themselves as schools for 
a particular kind of student or by otherwise encour- 
aging a certain kind of student to apply for admis- 
sion. A study by the U.S. Department of Education 
found that one quarter of charter schools are targeted 
toward certain populations (see Nelson, et. al. 2000). 
Many people familiar with charter schools would 
find this a very conservative estimate; as we will see, 
this impression is confirmed by our findings in this 
study. 

Targeted charter schools usually serve populations 
that are disadvantaged. Gronberg and Jansen find 
that Texas charter schools serve more black students, 
more non- Anglo students, and more at-risk students 
than regular public schools (see Gronberg and Jansen 
2001). Fuller and his co-authors attempt to downplay 
the differences between charter school student popu- 
lations and regular public school student populations 
by noting that the percentage of students in the fed- 
eral free and reduced limch program is similar at 
both kinds of schools. But the free and reduced lunch 
program serves a very broad population — it includes 
about 40% of all students. Because free and reduced 
lunch status does not differentiate between lower 
middle class students and students living in extreme 
poverty, it does not accurately identify the pool of 
students who are so poor that they are truly "at risk" 
educationally. Fuller and his co-authors themselves 
find that 44% of charter schools are identified as hav- 
ing a "special program focus" or as being an "alter- 
native school" (see Fuller, et. al. 2003). 

Because special-focus or alternative schools tend to 
target students with educational disadvantages, stu- 
dents at those schools t5q3ically do more poorly in 
school and perform worse on tests than the general 
student population. Gommon examples include 
schools for at-risk youth, disabled students, drop- 
outs, girls who are pregnant or have children, and 
juvenile delinquents. Some charter schools are also 



targeted at gifted or college-bound students, but 
these are much less common. 

The targeting of charter schools to disadvantaged 
populations is so common that many people have 
come to believe, incorrectly, that all charter schools 
serve disadvantaged students. Even some of the 
charter school staff members we spoke to in our 
phone survey (see below) shared this belief. One, 
when asked if her school was targeted to a particu- 
lar student population such as at-risk youth, replied, 
"well, that's what a charter school is."^ Another told 
us, "that's why charter schools were created in Texas, 
to serve at-risk kids."^ 

One reason many charter schools serve targeted 
populations is that the procedures by which new 
charter schools are created often encourage such tar- 
geting. These procedures vary considerably from 
state to state. In many cases, local school boards are 
given veto power over the creation of new charter 
schools within their districts, and even where they 
do not have veto power they can have other forms 
of influence over the process. This tends to discour- 
age the creation of charter schools that might effec- 
tively compete with regular public schools for 
students who are easier to educate, since school 
boards have no incentive to approve of the creation 
of such schools and every incentive to resist them. 
By the same token, the charter school approval pro- 
cess in most states tends to encourage the creation 
of charter schools targeted towards those students 
that regular public schools find less desirable because 
they are more likely to cause trouble, require extra 
academic help, or in some other way be difficult and 
expensive to educate. 

Of course it's to be expected that charter schools, like 
many school reform efforts, would emphasize help- 
ing disadvantaged students who need it the most. 
However, the existence of so many charter schools 
that serve targeted student populations is a serious 
obstacle to researching charter schools' academic 
performance. If charter schools have smaller year- 
to-year improvements in test scores than regular 
public schools, this does not necessarily tell us any- 
thing about charter schools' performance. It maybe 
that charter schools do a better job of teaching, but 
their performance on tests is less impressive because 
their students are more likely to face unusually se- 
vere obstacles to learning over which the school has 
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no control. Without more information about the stu- 
dent populations, we can't know. 

The lack of a valid comparison between charter 
school students and regular public school students 
is almost never adequately addressed in research on 
charter schools. The problem is often acknowledged, 
but few studies attempt to overcome it, and those 
that do make the attempt seldom have sufficient in- 
formation to do a good job of it. Most of the existing 
research on charter schools is therefore of little real 
value in evaluating the performance of charter 
schools. 

This point has begun to get some attention in the 
media. When charter school opponents in Illinois 
pointed out that charter schools were doing no bet- 
ter than regular public schools on state tests, 
Nezuszueek's Jonathan Alter wrote that "the compari- 
sons are meaningless" because "many of those char- 
ter schools are for troubled kids." Alter called this 
kind of false comparison a "smear" perpetrated by 
"mindless boards of education and reactionary teach- 
ers' unions" afraid of competition from charter 
schools (Alter 2002). 

But commentators like Alter still make up only a 
small minority of the public discussion of charter 
schools. Far more typical is Francis X. Clines, who 
pointed out in a recent New York Times op-ed article 
that nearly two- thirds of Texas schools rated as low- 
performing under initial implementation of the fed- 
eral No Child Left Behind law are "laissez-faire 
charter schools" (Clines 2003). Clines makes no men- 
tion of the enormous number of Texas charter schools 
that are targeted at at-risk youth and other educa- 
tionally disadvantaged populations. His readers are 
left with the clear implication that charter schools 
are failing to provide an adequate education. 

The root of the problem for performing good research 
on the academic effect of charter schools is that states 
do not make available sufficient information on the 
recruitment and admissions practices of charter 
schools, nor on the demographic profile of students 
that charter schools ultimately attract. Information 
on recruitment and admissions practices is supposed 
to be included in each school's charter, of course, 
but it isn't feasible for researchers to separately ob- 
tain and analyze the charter of every charter school 
in a state, much less in a broad sample of states. Re- 



searchers must therefore rely on information col- 
lected and made available by state education agen- 
cies. In some states, no information on charter school 
population targeting is made available; other states 
report some information, but not enough. For ex- 
ample, Arizona reports which schools choose to 
identify themselves on state paperwork as being tar- 
geted to specific populations, but targeted schools 
are not required to identify themselves in this man- 
ner, and even among those that do, many provide 
vague or incomplete information. As for demo- 
graphic information, the only data typically avail- 
able on both charter and regular public schools is 
student race and ethnicity. Information on free and 
reduced lunch status is usually available for regular 
public schools, but it is not available for most char- 
ter schools, and in any event the free and reduced 
lunch program is far too broadly inclusive to be use- 
ful in measuring truly at-risk student populations. 
With very different student populations being served 
in charter and regular public schools, and with vir- 
tually no demographic data that could be used to 
control for those differences statistically, it is ex- 
tremely difficult to conduct an appropriate evalua- 
tion of charter schools' academic quality. 

This problem has left researchers struggling for other 
ways to study charter schools. Of course, one way 
to solve the problem is simply to study something 
other than test scores, something for which student 
background differences are less likely to be relevant. 
For example, Vanourek, Manno, Finn, and Bierlein 
studied satisfaction levels by surveying students, 
parents, and teachers in charter schools on such 
matters as what they liked most and least about their 
school, how their school compares to public schools 
they have previously been associated with, satisfac- 
tion with academic performance, and the reasons 
they choose charter schools. The survey found high 
satisfaction with charter schools, which is hardly 
surprising since those who are dissatisfied with char- 
ter schools are likely to leave them, and thus not be 
included in a survey. However, the survey is useful 
in that it documents other reasons besides academ- 
ics that people seek out charter schools, such as bet- 
ter communication and responsiveness from school 
staff (see Vanourek, et. al. 1998). 

Other researchers have studied other kinds of out- 
comes for charter schools. Fuller and his co-authors 
find that charter schools have less money, fewer cre- 
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dentialed teachers, and fewer special-education stu- 
dents than regular public schools; that brand new 
charter schools and charter schools managed by pri- 
vate firms have fewer credentialed teachers; and that 
charter schools with more black students are less 
well-funded (see Fuller, et. al. 2003). A U.S. Depart- 
ment of Education phone survey of school district 
leaders in districts where charter schools are located 
found that nearly half report a budget impact from 
student transfers to charter schools, nearly half re- 
port that their districts have become more customer- 
service oriented, and most report that their districts 
have made educational changes in response to char- 
ter schools. In states where school districts are the 
only charter authorizers, or where districts are paid 
money when they lose students to charter schools, 
districts are less likely to report budget impacts or 
changes to education and customer service in re- 
sponse to charter schools. All districts where enroll- 
ment was declining prior to the opening of charter 
schools reported a budget impact, including layoffs, 
and all reported an increased focus on customer ser- 
vice as well as educational changes (see Ericson, et. 
al. 2001). 

A number of researchers have approached charter 
schools through case studies. Wells and her co-au- 
thors performed case studies of 17 charter schools 
in California; they argue that charter schools are "not 
held accountable" for performance by their autho- 
rizers, are not very innovative in their practices, and 
are not more cost-efficient than regular public 
schools. They also argue that the perception by regu- 
lar public schools that charter schools have unfair 
advantages "inhibits competition" (see Wells, et. al. 
1998). However, two U.S. Department of Education 
case studies that look at much broader samples of 
charter schools find more encouraging results. One 
study included field visits to 91 schools and con- 
cluded that charter schools are in fact held account- 
able for their performance, but not always by the 
same methods. The study also found that resource 
limitations are the biggest obstacle facing charter 
schools (see Nelson 2000). The other study included 
150 schools and 60 authorizing agencies, and found 
that charters learn quickly that the best way to sat- 
isfy their various constituents (parents, donors, com- 
munity leaders, district officials, etc.) is to focus on 
quality instruction. It also found that new types of 
charter authorizers (state boards of education, uni- 
versities, etc.) learn more quickly than do local school 



districts to break habits of accountability based on 
process compliance rather than on performance and 
outcomes (see Hill, et. al. 2001). 

In the end, though, there really is no substitute for 
knowing whether charter schools can make the grade 
academically. Some studies have compared charter 
school test scores to regular public school test scores 
without accounting for population differences at all. 
A recent study of California charter schools by the 
Charter Schools Development Center found that 
while charter schools overall had test scores below 
the average for regular public schools, charter 
schools more than five years old (a total of 80 schools) 
had scores above the average for regular public 
schools (see Agostini 2003). However, in the absence 
of information on student populations, it is not clear 
how much this descriptive statistic really tells us. 

Horn and Miron compare Michigan charter schools 
to their host school districts. Comparing individual 
schools to whole school districts is a very imperfect 
comparison, given the enormous demographic di- 
versity that is possible within a school district. This 
problem is somewhat mitigated because Horn and 
Miron look at year-to-year changes in test scores 
rather than the level of test scores in any one iso- 
lated year; year-to-year changes in test scores are less 
influenced by demographic factors. But the most 
severely at-risk students may still show worse year- 
to-year changes in performance than other students, 
so uncontrolled demographic differences are still a 
problem for Horn and Miron's study. 

Horn and Miron measure the difference between 
charter schools and host districts in two ways: the 
average score for all charter schools compared to the 
average score for all host districts, and the number 
of charter schools that outperform their host districts 
compared to the number of host districts that out- 
perform their charter schools. The second method 
gives us a limited amount of insight into the prob- 
lem of targeted charter schools: if charter schools are 
divided between well-performing schools serving 
general student populations and low-performing 
schools for at-risk populations, the average test 
scores might conceal this pattern, but looking at the 
number of schools that outperform their districts and 
vice versa might reveal it. Horn and Miron find that 
districts outperform charters in the aggregate, but 
their data tables reveal that in most grade levels and 
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test subjects, more charter schools outperform their 
host districts than vice versa (see Horn and Miron 
2000). Clearly, a minority of charter schools that may 
well be serving disadvantaged student populations 
dragged down the average charter school perfor- 
mance. Thus, Horn and Miron's study is seriously 
compromised by the incomparability of targeted 
charter schools and regular public schools. 

In studying California charter schools, Raymond also 
compares them to regular public schools in their host 
districts (as well as to the overall performance of 
regular public schools across the state). Raymond is 
able to cope with differences in student background 
better than Horn and Miron, because her study col- 
lects school-level data on poverty, race, parents' edu- 
cation, limited-English status, and testing inclusion. 
Using these data to provide demographic controls, 
Raymond finds that charter schools have test score 
improvements that are generally better than those 
of regular public schools, though the differences are 
only sometimes statistically significant (see 
Raymond 2003). 

One way for researchers to overcome the problem of 
incomparability between charter school populations 
and regular public school populations is to obtain 
specific demographic information on every individual 
student in the study. Obtaining a sufficient amount 
of such information is rarely feasible, but when it can 
be done researchers can draw valid comparisons be- 
tween charter schools and regular public schools. 
Using individual-level data on students in Arizona, 
Solmon, Paark, and Garcia were able to control for 
race, primary language, gifted or disability status, 
absenteeism, and prior year test scores. Although this 
does not include information on income and family 
status, and thus does not fully tell us whether stu- 
dents are "at risk" or not, the controls for race, lan- 
guage, disability, and absenteeism should do a decent 
job of allowing meaningful population comparisons. 
They found that students who were in their first year 
in a charter school had significantly lower reading 
scores than comparable students in regular public 
schools, but students who were in their second and 
third consecutive years in a charter school had sig- 
nificantly higher reading scores than comparable 
regular public school students. For math scores they 
found that charter school students had lower scores 
in their first year and higher scores in their second 
year, but they found no statistically significant differ- 



ence in their third year in a charter school (Solmon, 
Paark, and Garcia 2001). 

Another approach, useful when multiple years of in- 
dividual student test score data are available but in- 
dividual demographic data are not, is to use what 
statisticians call a "fixed-effects" model. This method 
keeps track of each student's performance separately, 
comparing a student's performance in one year to his 
performance in the next year, the year after that, and 
so on. By comparing each student only with himself, 
this method removes much of the influence of differ- 
ences in student populations. However, the tradeoff 
for this advantage is that more years of data are 
needed to make the comparisons meaningful. This is 
a particularly serious problem when studying char- 
ter schools, which in most cases have not existed for 
more than a few years. In addition, disadvantaged 
students may well exhibit slower year-to-year 
progress in addition to starting with lower test scores. 

Gronberg and Jansen use this method to study char- 
ter school students in Texas over three years. They 
conclude that students in charter schools officially 
designated as serving at-risk students saw larger test 
score gains than students in regular public schools. 
(During their study period, Texas had two catego- 
ries of charter schools, one category of schools that 
admitted exclusively at-risk students and another 
category that contained schools with various admis- 
sion policies; since then, all Texas charter schools 
have been converted to open-enrollment policies.) 
They found that the reverse was true for students in 
other charter schools, although this finding is some- 
what suspect because there were fewer schools of 
this type. More importantly, both these conclusions 
are tainted by the availability of only three years of 
data. Using other methods that do not address the 
problem of student population differences, they find 
that students tend to score worse after their first year 
in a charter school, but their performance compared 
to students in regular public schools recovers in sub- 
sequent years, and that charter schools are more cost- 
efficient than regular public schools (see Gronberg 
and Jansen 2001). 

It should be obvious that there is very little we can 
say with confidence about the academic performance 
of charter schools based on this research. Even those 
few studies that make some contribution to our un- 
derstanding are t 5 q)ically confined to only one state. 
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A study that includes only one state is always of lim- 
ited application to other states, and in the case of 
charter schools it may well have no application at 
all, given the large differences in the design and func- 
tioning of charter school systems in different states. 
We have been unable to find any previous multi- 
state study of charter school test scores that addresses 
the problem of student population differences be- 
tween charter schools and regular public schools. 

M ethod 

We included in our study eleven states — Arizona, 
CaUfomia, Florida, Texas, Michigan, Wisconsin, Ohio, 
Colorado, North Carolina, Minnesota, and Pennsyl- 
vania — that had large enough numbers of charter 
schools to give us a good chance of yielding suffi- 
ciently large samples after targeted schools were elimi- 
nated. We also included four other states — Illinois, 
New York, New Jersey, and Indiana — that had fewer 
charter schools but were of particular interest. Four 
of the states we examined — Illinois, New York, Indi- 
ana, and Wisconsin — were ultimately excluded from 
our analysis for lack of sufficient data.'* This left us 
with eleven states included in our analysis. 

We obtained a complete list of charter schools from 
each state, along with any available information on 
targeting of student populations in these schools. We 
then eliminated from these lists any schools that were 
known to be targeted to particular student popula- 
tions, along with any schools opened too recently to 
have test score data available for our study. (We could 
only examine schools that had two years' worth of 
test scores.) This gave us the list for our telephone 
survey. Calling every school on the list, we collected 
some basic information about each school (grades 
served, number of students, etc.) and then asked 
whether the school was targeted to a particular stu- 
dent population such as at-risk youth, dropouts, gifted 
students, or special education students. 

We considered a school targeted to an education- 
ally advantaged or disadvantaged population if it 
gave that population preferential treatment in ad- 
missions or if it made specific efforts to recruit that 
population, such as describing itself as a school for 
that type of student. We did not count a school as 
targeted if it had a distinctive curriculum or peda- 
gogy, such as the Montessori system or computer- 
based instruction, so long as it made no attempt to 



recruit or attract educationally advantaged or dis- 
advantaged populations in particular. We also did 
not count a school as targeted based solely on the 
local population in its location. For example, many 
charter schools told us that they had large num- 
bers of at-risk students; in such cases, we asked 
whether this was because the school targeted at- 
risk students or because the school just happened 
to get a lot of at-risk students as a result of its loca- 
tion. If the latter was the case, the school was not 
counted as targeted because it was serving the gen- 
eral student population in its area. Educationally 
disadvantaged student populations included 
groups such as at-risk students, pregnant or young- 
mother students, dropout recovery students, or dis- 
abled students, while educationally advantaged 
students included groups such as gifted or college- 
bound students. 

We excluded some charter schools from our study 
because they function in a way that does not allow 
them to be studied appropriately by our method. For 
example, we excluded "cyber" charter schools that 
provide instruction over the Internet, because the 
lack of a central campus to which students must 
travel leaves us without a real physical location for 
the school. This makes it impossible to compare that 
school with the closest regular public school (see 
below). We also excluded "conversion charters" — 
regular public schools that converted into charter 
schools — if they retained their assigned neighbor- 
hoods after the conversion. Students in those neigh- 
borhoods are automatically assigned to the 
converted charter school unless they specifically re- 
quest to be transferred to a non-charter school. We 
found 235 such schools in California, and a handful 
in several other states. We excluded these schools 
because they are not operating as schools of choice 
like other charter schools. 

Once we had completed our phone survey, we had 
a list of charter schools in each state that we had con- 
firmed were serving general student populations. We 
then obtained from each state a complete list of regu- 
lar public schools. Using a street atlas computer pro- 
gram, we mapped out the location of every regular 
public school and every charter school in each state. 
For each test score we had from a given charter 
school, we found the closest regular public school 
for which we had test scores from the same grade. 
We ignored regular public schools known to be tar- 
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geted to particular populations, such as magnet 
schools and schools for juvenile delinquents, as well 
as regular public schools for which we had no test 
scores comparable to the test scores we had for our 
charter schools. 

Comparing charter school test scores to the test 
scores of their closest regular public school pro- 
duces a much more fair comparison than, for ex- 
ample, comparing charter schools to their school 
districts. A given school district will normally con- 
tain a wide variety of student populations. In many 
cases, the same school district might contain both a 
depressed urban area and a booming suburban 
area. If it is the case, as we have good reason to 
believe, that charter schools are more likely to be 
located in the poorest parts of their districts, then 
comparing each charter school to its district will not 
be a fair comparison, as the charter schools are serv- 
ing a much more disadvantaged population. How- 
ever, the closest regular public school to a given 
charter school can reasonably be expected to serve 
a similar population. To be sure, their student popu- 
lations will not be exactly the same, but they will 
be much more similar to one another than any other 
available comparison. In this way, we used geog- 
raphy to control for demography. 

Rather than look at each school's test score levels in 
a single year, we looked at year-to-year test score 
changes. Specifically, we looked at the change in each 
test score for the two most recent years. For example, 
if the most recent available year for 3rd grade math 
scores was 2002, we looked at the change in 3rd grade 
math scores between 2001 and 2002. We used each 
school's average scale score or percentile rank, ex- 
cept in Michigan and Ohio where only the percent- 
age of students at each school who passed the test 
was available. In Michigan, test scores from the first 
and second years included in our study were not di- 
rectly comparable because of changes in the testing 
system, so we standardized the test scores to make 
them comparable. 

Looking at year-to-year score changes rather than 
single-year score levels allows us to further filter out 
some of the influence of student and family back- 
ground factors, focusing instead on the contribution 
each school makes to learning. It is well established 
that students from more advantaged backgrounds, 
who have had more opportunities to learn and face 



fewer obstacles to learning, score better on standard- 
ized tests. However, the year-to-year improvement 
(or lack thereof) that each student makes is more 
directly influenced by the quality of the school. This 
is sometimes called "value added" analysis, since it 
is a more direct measurement of the educational 
value added to a student by his school rather than 
the baseline from which the student started. Since 
the goal of our study is to overcome the problem of 
differences in student populations, it is particularly 
important that we remove the influence of student 
background characteristics to the extent that we can. 
Excluding targeted charter schools that are imlikely 
to have a counterpart in the regular public school 
system is by far the most important way in which 
we do this; comparing each charter school to its clos- 
est regular public school is another. Using year-to- 
year score changes instead of single-year score levels 
further refines our research design. 

In each state, we ran a regression analysis on year- 
to-year score changes for each test subject (math, 
reading, language, or science). This analysis con- 
trolled for whether the school was a charter school, 
in order to determine what effect charter schools 
have on test scores as compared to their closest regu- 
lar public schools. We also included a control vari- 
able for race (specifically, the percentage of the 
student body that is white, according to data from 
the National Center for Education Statistics) and a 
control variable for each tested grade level.® 

We report results in each of the states where we 
found enough untargeted charter schools to be con- 
fident that our results were meaningful. The results 
reported for each state have been standardized to 
allow for accurate state-to-state comparisons (Table 
containing these results is located in the Appendix). 
In the table, the first column gives the standardized 
coefficient representing the effect that untargeted 
charter schools had on test scores. A positive num- 
ber means charter schools had greater year-to-year 
test score improvement than their closest regular 
public schools, while a negative number means char- 
ter schools had smaller year-to-year test score im- 
provement. Results that are statistically significant 
at a high level (p<0.05), meaning we can have very 
high confidence they reflect the real effect of 
imtargeted charter schools, are marked with double 
asterisks, while results that are statistically signifi- 
cant according to a lower standard (p<0.1) are 
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marked with single asterisks. Results not marked 
with asterisks are not statistically significant. 

The figures in the first column represent the change 
in test score as measured in standard deviations. For 
example, untargeted charter schools in Texas made 
0.18 standard deviations greater progress on TAAS 
math scores than neighboring public schools. The 
figures in the second column translate these num- 
bers into the percentile point change this would rep- 
resent for a student who started at the 50th 
percentile — that is, exactly in the middle. Returning 
to our example, an improvement in TAAS math 
scores of 0.18 standard deviations would be equiva- 
lent to an improvement of 7 percentile points for a 
student who started at the 50th percentile. In other 
words, it would be the equivalent of a student who 
originally scored at the 50th percentile improving 
his test score to the point where he scored at the 57th 
percentile. Students starting either above or below 
the 50th percentile would see a correspondingly 
smaller change in percentile point terms from the 
same standard deviation change in test scores. 

We also combined the data from all eleven states and 
performed a national analysis. In addition to con- 
trolling for race and grade level, the national analy- 
sis controlled for each state included in the study. 
This analysis gives us a picture of the effect that 
untargeted charter schools have on test scores across 
the country. 

Results 

Populations Served by Charter Schools 

One thing we wanted from our study was simply to 
get a rough picture of how frequently charter schools 
were targeted to educationally advantaged or dis- 
advantaged student populations. We do not have a 
perfectly representative picture here, since we do not 
have data on schools that did not respond to our tele- 
phone survey, or that were not called because they 
were too new to have test scores available. In par- 
ticular, some states provided a lot of up-front infor- 
mation on which schools were targeted, while others 
provided little information, and still others no in- 
formation at all. This means that our telephone sur- 
vey method was not precisely the same in every state; 
in some states we began by excluding large num- 
bers of schools from our survey, in others fewer 



schools, and in some states we had to call each and 
every charter school in the state. However, by look- 
ing at those schools for which we do have data, we 
can get at least a rough picture of what populations 
are being served by charter schools. 

Results for targeting of charter schools to particular 
populations are presented in Table 1. We found very 
large variations from state to state in the targeting 
of charter schools. In Michigan, 95% of charter 
schools for which we have data were untargeted, 
whereas in Wisconsin 12.0% of charter schools for 
which we have data were untargeted. Other states 
ran the gamut between these extremes. There was 
also variation in the presence of charter schools tar- 
geted at advantaged populations; in Arizona 9.2% 
of schools for which we have data were "targeted 
high," while in four states we found no such schools 
whatsoever. In California we found a very large 
number of so-called "conversion" schools that had 
to be excluded from our study, and in Pennsylvania 
we found a significant number of "cyber" schools. 
Those excluded schools are represented in the 
"other" column. 

National Charter School Effect 

The results of our national analysis were drawn from 
the combined data for all eleven states. They repre- 
sent the effect that all the untargeted charter schools 
in our study had on test scores when compared to the 
performance of their closest regular public schools. 

These results showed a positive effect from charter 
schools and were statistically significant, but the size 
of the effect was modest. Untargeted charter schools 
made math test score improvements that were 0.08 
standard deviations greater than those of neighbor- 
ing public schools during a one year period. For a 
student starting at the 50th percentile, this would 
amount to a gain of 3 percentile points, to the 53rd 
percentile. Reading test score results showed 0.04 
standard deviations greater improvement in 
untargeted charter schools than in their closest regu- 
lar public schools over the course of a year, a benefit 
that would raise a 50th-percentile student 2 percen- 
tile points to the 52nd percentile. 

Because these results are statistically significant, we 
can be very confident that the charter schools in our 
study did have a positive effect on test scores. How- 
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ever, the small size of this effect should caution us 
against too much enthusiasm regarding the benefits 
of charter schools. The results of our study strongly 
support the conclusion that untargeted charter 
schools are somewhat better than regular public 
schools serving similar populations, but not a great 
deal better. 

Arizona Charter School Ejfect 

Arizona gave two statewide standardized tests. The 
Stanford Achievement Test, Ninth Edition (SAT-9), 
a nationally normed test used in many states, was 
given in grades 2-9 (the reading test was given in 
grades 1-9). Arizona's Instrument to Measure Stan- 
dards (AIMS), a state test geared to Arizona's school 
curriculum, was given in grades 3, 5, 8, and 10-12. 

Our Arizona results found weak and mixed effects 
from charter schools, and none that was statistically 
significant. We found small positive effects on the 
SAT-9 and small negative effects on the AIMS, but 
because none of these results was statistically sig- 
nificant we cannot be confident that either of these 
effects was really caused by charter schools. On the 
SAT-9 we found that charter schools had made 
progress of 0.03 standard deviations higher in math 
scores and 0.02 standard deviations higher m read- 
ing scores than that of regular public schools in one 
year's time, each of which is equivalent to 1 per- 
centile point of improvement for a student starting 
at the 50th percentile. They also had made progress 
in language scores over the course of one year that 
was 0.01 standard deviations greater than that of 
regular public schools, equivalent to less than 1 
percentile point of improvement. On the AIMS we 
found that charter schools had made less improve- 
ment in math scores by 0.06 standard deviations, 
equivalent to a drop of 2 percentile points for a stu- 
dent starting at the 50th percentile, and less 
progress in reading scores by 0.03 standard devia- 
tions, equivalent to a drop of 1 percentile point from 
the 50th percentile. 

Given that the direction of the untargeted charter 
school effect (positive or negative) was mixed for dif- 
ferent tests, the magnitude was always small, and 
none of the results was statistically significant, it is 
difficult to reach any conclusions about Arizona 
charter schools. We cannot be confident that they 
have any effect at all on test scores. 



California Charter School Effect 

California gave the SAT-9 in grades 2-11. Our Cali- 
fornia results found effects that were somewhat 
larger than those in Arizona, and in this case all the 
effects were positive. However, our results still were 
not statistically significant. We found that untargeted 
charter schools had math test score gains over a one- 
year period higher than those of nearby regular pub- 
lic schools by 0.06 standard deviations, and reading 
test score gains higher by 0.05 standard deviations, 
each of which is equivalent to a 2 percentile point 
gain for a student starting at the 50th percentile. 
Untargeted charter schools also had language test 
score gains 0.07 standard deviations higher than 
those of regular public schools, equivalent to a 3 
percentile point gain from the 50th percentile. None 
of these results was statistically significant, so al- 
though these results are more positive than our Ari- 
zona results they still do not justify strong 
conclusions about the effect of imtargeted charter 
schools in California. 

Florida Charter School Effect 

Florida gives two statewide standardized tests in 
grades 3-10. They are the SAT-9 and Florida's Com- 
prehensive Assessment Test (FCAT), a state test 
geared to Florida's school curriculum. 

Our Florida results yielded stronger positive effects 
from untargeted charter schools than our Arizona 
and California results, and two of these findings were 
statistically significant. We found that charter schools 
achieved year-to-year SAT-9 math score gams and 
FCAT reading score gains that were each 0.15 stan- 
dard deviations greater than those of nearby regu- 
lar public schools, equivalent to a gam of 6 percentile 
points for a student starting at the 50th percentile. 
Both these results reached a high level of statistical 
significance, so we can be very confident that the 
charter school effects were positive. We also found 
that charter schools had greater annual improvement 
on SAT-9 reading scores and FCAT math scores by 
0.08 and 0.12 standard deviations, respectively, equal 
to gains of 3 and 5 percentile points from the 50th 
percentile. However, these two findings were not 
statistically significant. 

These findings allow us to reach a firm conclusion 
regarding Florida charter schools. We can be very 
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confident that charter schools that serve the general 
population of students experienced greater gains in 
SAT-9 math and FCAT reading scores than did 
neighboring regular public schools. While still mod- 
erate, these gains were larger than those in Arizona 
or California. 

North Carolina Charter School Effect 

North Carolina gives a state-designed "end of grade" 
exam in grades 3-8. Our North Carolina results were 
the weakest of the five states for which we report 
separate results. We found that untargeted charter 
schools had year-to-year math score gains that were 
0.02 standard deviations higher than those of regu- 
lar public schools, equal to a 1 percentile point gain 
for a student starting at the 50th percentile, and read- 
ing score gains that were 0.01 standard deviations 
higher, equal to less than a 1 percentile point gain. 
Neither of these findings was statistically significant, 
so we cannot be confident that untargeted charter 
schools in North Carolina had any effect on test 
scores. 

Texas Charter School Effect 

Texas gave the Texas Assessment of Academic Skills 
in grades 3-10. Our Texas results were the strongest 
in our study. We found that untargeted charter 
schools in Texas achieved math score improvements 
over a one year period that were 0.18 percentile 
points greater than those of neighboring public 
schools, equivalent to a gain of 7 percentile points 
for a student starting at the 50th percentile. We found 
that Texas charter schools produced a reading score 
benefit of 0.19 percentile points, equivalent to a gain 
of 8 percentile points from the 50th percentile. Both 
these findings were statistically significant at a high 
level. These findings suggest that Texas charter 
schools that are not targeted toward educationally 
advantaged or disadvantaged populations are able 
to make significantly greater annual progress on 
standardized test scores than their regular public 
school counterparts. 

Conclusion 

We found positive effects from charter schools serv- 
ing general populations, but for the most part these 
effects were modest in size. This suggests that while 
charter schools benefit from their partial indepen- 



dence from the regular school system, they still face 
significant obstacles to more effective reform. 

There are several possible explanations for the find- 
ing that charter schools outperformed their regular 
public school counterparts. One possibility is that 
charter schools benefit from the freedom they enjoy 
from many state regulations. With less of a regula- 
tory burden, charter schools may be able to focus more 
of their energies on assisting students and enjoy 
greater flexibility in meeting student needs. The bet- 
ter academic performance of charter schools could also 
be a result of their being schools of choice. By allow- 
ing families to send their children to schools of their 
own choosing, charter schools may permit more effi- 
cient matching of student needs with school capabili- 
ties. Given that not every child learns in the same way, 
charter schools help families find a school that works 
well for them and their children. This "choice factor" 
may also produce better academic performance in 
charter schools because of the incentives that charter 
schools face when attempting to attract and retain stu- 
dents. Unlike public schools, to which students are 
assigned based on residence, charter schools have to 
earn their students. This makes it more likely that they 
will better serve those students' needs. 

One possible explanation for the modest size of posi- 
tive charter school effects is that charter schools 
cally receive less money than regular public schools. 
The per-pupil allocation charter schools receive for 
operating expenses is often somewhat less than regu- 
lar public schools receive, and charter schools almost 
always receive significantly less money than regu- 
lar public schools — or no money at all — to cover capi- 
tal costs such as construction and maintenance. 
However, it is unlikely that this is the most impor- 
tant explanation, because our results do not show a 
clear relationship between funding of charter schools 
in each state and charter schools' effects on test 
scores. For example, while the state with our weak- 
est test score results. North Carolina, provides no 
capital funding at all for charter schools, the same is 
true of Texas, the state with our strongest results.^ 

Another possible explanation is that charter schools, 
though exempt from many regulations, are still pub- 
lic schools subject to significant regulatory burdens. 
These will vary considerably from state to state, of 
course. In many states, charter schools are not auto- 
matically exempted from all relevant state regulations. 
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but instead must apply for the specific exemptions 
they want. Different states give charter schools dif- 
ferent degrees of latitude in curriculum, hiring, fir- 
ing, organizational structure, and budgeting. A recent 
article in the Los Angeles Times detailed the significant 
regulatory burdens on charter schools in CaUfomia 
(see Mathews 2003). Also, depending on its structure, 
the charter granting process may discourage bolder 
innovations in favor of schools that will look more or 
less like the status quo. Furthermore, charter schools 
are subject to the oversight powers of their charter 
authorizing agencies. In places where these powers 
are generously defined and aggressively wielded, the 
real independence of charter schools may be limited. 
If the regulatory burden imposed on charter schools 
is not all that different in practice from the burden on 
regular public schools, it would not be surprising to 
find only modest differences in how these different 
types of schools perform academically. 

One other possible factor is the requirement in many 
states that charter school students take standardized 
tests specifically geared to state curricula. Part of the 
possible benefit that charter schools might provide 
is through curriculum innovation; if schools are 
judged by student performance on a test designed 
for the state's curriculum, they will not be as free to 
experiment with curriculum changes. However, this 
point would not apply to nationally norm-referenced 
tests such as the SAT-9, which are broader measures 
of student performance. 



Finally, the most important factor limiting the per- 
formance of charter schools may well be their new- 
ness. Charter schools have not been aroimd for a very 
long time, and in most states they are overwhelm- 
ingly brand new schools rather than converted 
schools. It takes a while for a new school to get its 
house in order — to figure out what works and what 
doesn't. It may be that as charter schools get older, 
their performance compared to regular public 
schools may become more impressive. 

While this study's research design does a better job 
than previous designs at screening out the effects of 
charter school targeting and drawing fair compari- 
sons between charter and regular public schools, it 
is still possible to do better. A random assignment 
study of charter and regular school students would 
go even further toward establishing a fair compari- 
son. Random assignment, which compares the per- 
formance of two randomly-selected groups that are 
the same in almost every way except for the factor 
being studied, is the "gold standard" for social sci- 
ence research. A study comparing students who did 
and did not get admitted to charter schools through 
a random lottery would provide the best possible 
comparison between charter and regular public 
schools. Every effort should be made to design and 
implement a random assignment study of charter 
school performance to increase the confidence with 
which we can draw conclusions about their academic 
effectiveness. 
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Endnotes 

1. Data obtained from the U.S. Charter Schools website, sponsored by the U.S. Department of Educa- 
tion ( www.uscharterschools.org ). 

2. Phone interview with a charter school staff member (January 16, 2003, 1:35 pm). 

3. Phone interview with a charter school staff member (January 28, 2003, 10:43 am). 

4. Indiana was excluded because all 13 of its charter schools are too new to have test score data avail- 
able. New York was excluded because the state was only able to provide test score data in the two most recent 
years for five schools, too few to produce meaningful results. We had already conducted our phone survey in 
New York when we discovered this, so we do have information on the targeting of New York charter schools. 
This information is presented in Table 1. After excluding targeted schools, Illinois and Wisconsin lacked a 
sufficient number of remaining charter schools to warrant further test score analysis. 

5. Data in our regression analysis were weighted based on each school's enrollment. 

6. North Carolina funding information obtained via interview with Otho Tucker of the state's educa- 
tion department (April 15, 2003, 3:10 pm); Texas funding information obtained from the state's website 
( http:/ /www. tea. state, tx.us /charter /faqfunding.htm ). 
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Appendix 



Table 1: Populations Served by Charter Schools 





1 Untargeted 


Targeted Low 


Targeted High 


Other 


Arizona 


47.5% 


42.3% 


9.2% 


0.9% 


California 


18.6% 


3.6% 


1.0% 


76.8% 


Colorado 


67.3% 


25.0% 


5.8% 


1.9% 


Florida 


57.6% 


37.4% 


0.7% 


4.3% 


Illinois 


25.0% 


66.7% 


8.3% 


0.0% 


M ichigan 


95.0% 


2.8% 


0.7% 


1.4% 


M innesota 


38.8% 


57.1% 


4.1% 


0.0% 


NewJ ersey 


74.2% 


19.4% 


6.5% 


0.0% 


N ew York 


78.3% 


13.0% 


0.0% 


8.7% 


North Carolina 


84.9% 


12.3% 


2.7% 


0.0% 


Ohio 


65.2% 


33.3% 


0.0% 


1.4% 


Pennsylvania 


69.1% 


18.2% 


0.0% 


12.7% 


Texas 


58.6% 


41.4% 


0.0% 


0.0% 


Wisconsin 


12.0% 


78.7% 


4.0% 


5.3% 



Table 2: Charter School Effects 





Charter Effect 


Change from 50th Percentile 


N umber of 


National 


Math 


-fO.08** 


-f 3 percentile points 


2,126 


Reading 


-fO.04** 


-f 2 percentile points 


2,388 


Arizona 


SAT-9 M ath 


-F0.03 


-Fl percentile point 


641 


SAT-9 Reading 


-F0.02 


-Fl percentile point 


643 


SAT-9 Language 


-FO.Ol 


Less than 1 percentile point 


637 


AIMS Math 


-0.06 


-2 percentile points 


316 


AIMS Reading 


-0.03 


-1 percentile point 


305 


California 


SAT-9 M ath 


-F0.06 


-f 2 percentile points 


175 


SAT-9 Reading 


-F0.05 


-f 2 percentile points 


174 


SAT-9 Language 


-F0.07 


-f 3 percentile points 


171 


Florida 


SAT-9 M ath 


-F0.15** 


-f 6 percentile points 


293 


SAT-9 Reading 


-F0.08 


-f 3 percentile points 


293 


FCAT Math 


-FO.12 


-F5 percentile points 


286 


FCAT Reading 


-F0.15** 


-f 6 percentile points 


286 


North Carolina 


End of Grade Math 


-F0.02 


-Fl percentile point 


368 


End of Grade Reading 


-FO.Ol 


Less than 1 percentile point 


364 


Texas 


TAAS Math 


-F0.18** 


-F7 percentile points 


319 


TAAS Reading 


+0.19** 


-f 8 percentile points 


319 



** = statistically significant at p< .05; * = statistically significant at p< .10 
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publishes studies, and holds luncheon forums where prominent local and national leaders are given opportunities 
to present theirviews on critical urban issues. Cities on a Hiii, CCI's newsletter, highlights the ongoing work of 
innovative mayors across the country. 

The M anhattan Institute is a 501(C)(3) nonprofit organization. Contributions are tax-deductible to the fullest 
extent of the law. BIN #13-2912529 
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